DOCUMENT RESUME 



ED 053 087 



24 



SP 005 130 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 

BUREAU NO 
PUB DATE 
CONTRACT 
NOTE 



Borg, Walter R. 

The R & D Process as Used in Designing Minicourses. 
Far West Lab. for Educational Research and 
Development, Berkeley, Calif. 

Office of Education (DHEW) , Washington, D.C. Bureau 
of Research. 

BR-6-2931 

70 

OEC- 4-7- 06293 1-30 64 



1 2 p. 



EDSS PRICE 
DESCRIPTORS 



IDENTIFIERS 



EDRS Price MF-S0.65 HC-J3.29 

Costs, ^Educational Development, Educational 
Research, ^Instructional Materials, *Material 
Development, Microteaching, Short Courses, ^Teacher 
Education 
♦Miuicourses 



ABSTRACT 



This paper elaborates on a list of procedures 
contained in an accompanying handout entitled "The 27 Steps in the 
Development Program.” These 27 steps are subsumed under 12 
categories: 1) research and information collecting, 2) planning, 3) 
develop preliminary form of product, 4) preliminary field testing, 5) 
main product revision, 6) main field testing, 7) operational product 
revision, 8) operational field testing, 9) final product revision, 

10) dissemination and distribution, 11) report preparation, and 12) 
implementation. The paper emphasizes the importance of detailed 
planning with specific objectives. It also justifies the expense of 
such rigorous development by pointing out the wide applicability of 
the products and the resulting low unit cost per user. (RT) 
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THE R & D PROCESS AS USED IN DESIGNING MINICOURSES 
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Walter R. Borg 

Far West Laboratory for Educational Research and Development 



In recent years, education has finally developed a reasonably adequate 
technology. However, at the present time few people in education have any 
understanding of the educational R & D process and fewer still have ever 
applied the process to an educational problem. Many educators still seem 
to vaguely link educational technology with hardware. In fact, educational 
R \ D is a systematic process- that can be followed in des.igaing, building 
and evaluating an educational product. Although most educational products 
involves a mix of both human and nonhuman resources, the presence of non- 

-■ — —* *■ H 

human resources such as computers, videotape recorders and the like, is 
in no way essential to the technological process. 

The Teacher Education Program that I have directed at the Far West 
Laboratory for Educational Research and Development has built an educational 
technology that has proven to be effective in the design, development, 
evaluation, and improvement of educational products. I am here today to 
describe this technology to you and report some of our experience in 
applying it to the development of specific teacher education materials. 

The educational products that we have been developing at the Far West 
Laboratory are called Mini courses. A Mini course is a carefully tested and 
validated educational package designed to help teachers improve specific 

I 

instructional skills. The typical Minicourse deals with about a dozen 

f 

highly specific teaching skills. During the 15 hours required to complete 

/ 

* Paper presented at the National Conference of Regional Laboratory and 
R & D Center Directors. San Diego, California, December 1970. 
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a Minicoirse, the teacher devotes about 4* hours to viewing films which 
give a precise definition of each skill and show numerous classroom 
examples of how the skill may be used in regular teaching situations. 

The remaining 11 hours are devoted to carefully structured teacher 
practice in simplified teaching situations and self-evaluation and improve- 
ment based upon the teacher's analysis of videotape recordings of his own 
practice lessons. 

I believe it will help you to understand the development process. that 
has evolved in the Minicourse program if you first have a clear idea of 
the products that we are building. Therefore, I would like to show you 
at this time an 8 minute film which is used to introduce the teachers to 
our first Minicourse. This Minicourse deals with twelve specific skills 
related to**improving the teachers' effectiveness in carrying out a discussion 
lesson. (Show the Minicourse 1 introduction film at this time.) You can 
get an idea of the scope of our program by referring to the handout 
entitled Mini courses Currently Being Developed . 

Now that you have some idea of the kinds of educational products that 
we are developing, I would like to review with you the specific educational 
technology that we employ in developing the Minicourses. Let me call your 
attention to the handout you have received entitled The 27 Steps in the 
Development Program . This paper briefly outlines our development process 
and I will refer to it in describing the various steps involved in that 
process. 

Once we have tentatively identified an instructional area in which a 
Mini course appears to be needed, our first step is to carry out an exhaustive 
review of the literature related to this area. The review of the literature 



3 



has several purposes. First, of course, it gives us a clear picture of 
the state of knowledgej a n the area in which we have chosen to work. 
Secondly, it provides us with the information we need to clarify our own 
objectives and carry out the initial planning necessary before moving into 
development of the product itself. Finally, it identifies gaps in the 
current’ stat^ of knowledge. We sometimes find' it necessary to carry out 
small scale research projects to at least partially fill the more critical 
gaps that we discover. 

At this point it seems appropriate to acquaint you with one of the 
facts of life that one learns very rapidly when carrying out educational 
development. This fact is that; the amount of solid research evidence that 
we have on any subject in education is not likely to be a sufficient foun- 
dation for" technological development of a new product or process. Further- 
more, there are usually so many gaps in our knowledge that if we delay 
technological development until a firm and complete foundation of research 
has been built, we will delay for a very long time. Therefore, development 
in education at this time must be built on a combination of research 
evidence, the insight and knowledge of practi oners in the field, plus a 
large dose of common sense. 

In planning the development of a new educational process or product, 

I would emphasize two points. The first is that your initial planning 
should be as thorough and detailed as possible even though it is likely 
most of your plans will eventually be changed in the light of experience 
gained during the actual development and evaluation of your product. 

Perhaps the most important advantage of a detailed and carefully thought 
out initial plan is that it gives you a framework upon which changes can 
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be made ab the need for change becomes apparent. A thorough plan is also 
much more useful in hel (Dijig. you identify places where changes are needed. 
Sketchy plans usually result in a waste of money and time that could have 
been avoided if you had adequately thought through your project in advance. 

The second point I would make about planning is to emphasize the great 
importance of'bui'lding very specific objectives' to be achieved by the pro- 
duct or process that you plan to develop. If the product you are building 
is concerned with teaching behavior as is the case with Our Mini courses, 
you should spell out as thoroughly as possible the specific behaviors that 
teachers will be able to display as a result of completing the training. 

Let me give you an example of a specific objective for Minicourse I. This 
objective states: ‘‘Given a discussion lesson, teachers who complete Mini- 

course 1 wfll ask questions which, in at least 50 percent of the cases, 
will require students to use higher cognitive processes rather than 
recitation of facts. In order for the course to be successful, this ob- 
jective will be achieved by at least 75 percent of the teachers taking the 
course." Although such objectives are difficult to build and may well have 
to be changed as we learn more about the teaching and learning process, 
they are still extremely important in technological development. Such 
objectives provide definite goals which are to be achieved as the product 
progresses through field testing and evaluation. They also provide the 
developer with a clear definition of failure. A great advantage of a 

rigorous development process is that it provides evidence of success or 

\ 

failure. In contrast, most processes and products now in use in education 
are used year after year without anyone knowing whether they achieve their 
objectives. 

O 
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Once planning has been completed, the next step in the technological 
development of an educational product is to build a prototype. This pro- 
totype should be built at a minimum cost and with a minimum of unnecessary 
embellishments. However, the product should be as good as you are able to 
build it in its essential elements. Thus, we know that simple clear 
i 1 1 ustra'tions' tend to help the learning process'. We also know, however, 
that beautifully done color illustrations are of no more value to learning 
than crudely done black and white illustrations provided only that the 
latter are clear and understandable. 

Once we have developed the preliminary form of our educational product, 
the next step is to take it to the field and try it out in as realistic 
conditions as possible. Educational products must be evaluated in terms 
of their effectiveness under field conditions. Therefore, Laboratory data 
or data collected under highly artificial field conditions are of limited 
value. We generally look for two things in field testing a Minicourse. 

One is'the perceptions of the user. His perceptions of the product are 
very important since a product which is disliked by the user has very little 
chance of making any real change in the educational process. The second 
thing we evaluate, of course, is the degree to which the product meets its 
objectives. To bring about educational improvement, a new product or 
process must meet both of these conditions. 

The feedback that we obtain from field testing is, of course, brought 
back into the Laboratory and used as a basis for revising the Minicourse. 
This revision is again tested in the field and feedback and evaluation 
data are again obtained. This cycle of field test, evaluate, revise 
should be continued until the product meets its educational objectives 
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and is perceived favorably by most use^s.' You will note on the handout 
that Minicourses typically undergo three field tests and revision. Each 
field test in our cycle of technological development has slightly different 
purposes but the basic purpose, that is, evaluating the product's effective- 
ness and getting feedback for further improvement is essentially the same. 

We have found 'three field tests to be sufficient for most Mini courses 
although on a few occasions a fourth test has been necessary in order to 
bring the course up to the point where its educational objectives are. met. 
The final steps in the development of Mini courses are aimed at preparing 
the course for widespread .use, disseminating the course and assisting 
with its implementation in the schools. At present, we feel much less 
confident of our ability to disseminate our courses and bring about their 
widespread use than we do in our ability to develop courses that bring 
about substantial improvements in teaching. Our courses are currently 
being released by a commercial publisher, but I do not really know whether 
this is the best approach to achieve widespread implementation. The ad- 
vantages of the commercial publisher are, briefly, that such publishers 
are experienced in widespread product dissemination and have a corps of 
trained salesmen in the field who are already in contact with educators. 
Surely, the biggest disadvantage of the commercial publisher is that the 
cost of an educational product goes up a great deal when it is distributed 
through commercial channels. 

I would like to make one additional point concerning the use of a 
rigorous research and development process, that is, this type of work 
is difficult, time consuming, and expensive. The R & D cycle we follow 
requires field testing and revising each Minicourse at least three times 
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and provides for the collection of pre- and post-course performance data 

to determine if teachers* taking. the course can perform the skills in their 

regular classrooms. A typical minicourse requires about 18 months to 

carry through our complete R & D cycle. During this time about seven 

thousand man-hours of effort are expended on the course. The average cost 
* * * 

of developing a Mini course has been nearly $107,000. Of this amount, 
review of the literature and planning has cost about $8,000. Carrying out 
three field tests and evaluating the results of these tests for each course 
has cost an average of $40,000. Developing the initial form of the course 
and making major revision's after the field tests has cost approximately 
$52,000. The remaining $7,000 has been devoted to preparation of final 
reports and expenses incurred in dissemination and implementation. How- 
ever, it should be remembered that most of the dissemination and imple- 
mentation costs are borne by the commercial publisher. The cost to the 
publisher of taking a fully developed and tested Minicourse, producing this 
course commercially and putting it on the market has been about $50,000 per 
course. 

In view of these costs, you may ask: Is rigorous technological develop 

ment of educational products really feasible from an economic standpoint? 

I would assert most strongly that it is for two important reasons. First, 
this type of development leads to real improvement in education, that is, 
a product developed following the process that we use at the Far West 
Laboratory is demonstrably superior and, since it involves training 
teachers, can be expected to have a long-range effect on the learning of 
a great many children for each teacher who is trained. The second reason 
that I regard such development to be a wise educational investment is that, 
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although the cost of developing an educational product is high, if such 
products are broadly implemented, the unit cost per user of such develop- 

* a 

• t 

ment is small. If we take our first Mini course as an example, there are 
about 614,000 teachers in elementary schools in the United States who 
are at the appropriate grade levels for this course (Simon & Grant, 1968). 
If one* out of every ten- teachers takes this course, the development cost 
per teacher will be only $1.63. And in the case of Minicourse 1, there 
is a good chance that the federal money spent to develop' this course will 
be returned to the U. S. Treasury. In fact, if the commercial publisher's 
market research is correct, the Treasury will receive royalties that exceed 
our development costs by about $160,000*. Thus, although the research- 
based development of educational products is expensive, this process is 
by far the**most economical that we now have for bringing about significant 
nationwide improvements in educational practice. 

In addition to cost-effectiveness, a rigorous development approach 
has several other advantages over both local innovations and research. 
First, it provides the educational practitioner with a product that is 
fully ready for operational use. Therefore, implementing an educational 

i 

product that has undergone development is much easier and much less likely 

' * 

to fail than implementing an innovation such as team teaching in which 
each user must re-invent most of what he needs to put his program into - 
effect. ' 

Second, a product of educational development carries with it rigorous 
research evidence that it does the job for which, it was designed. For 

* Minicourse 1 is being marketed by Macmillan Educational Services, Inc., 

8701 Wil shire Boulevard, Beverly Hills, California 
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example, to test the effectiveness of ouj* Mini courses , we make videotape 
recordings of the teacher's actual classroom performance before and 



ments in the teaching skills covered in the course, we continue to revise 

and field test the course until such gains are obtained. Only then do we 

release the course for pperational use in inservice training programs. 

Thus, when the educator adopts a product that has been developed by the 

R & D process, he can expect an improvement in his educational program, 

not just a change. Since education is such a huge enterprise, even small 

educational improvements, when applied widely, can bring about cost effective- 

• • <*• 

ness gains worth millions of dollars. If, through educational development, 
we can put together enough small improvements we can bring about a quiet 
revolution»in our schools. 

Another important advantage of educational development is that it 
provides an effective means of enlisting both local innovation and educa- 
tional-research in the task of building a better educational system. For 
example, if a promising local innovation is subjected to the development 
process and proves to be effective, it can become a lasting national 
improvement instead of a passing local fad. Furthermore, the research 
knowledge that in the past has made little impact on educational practice 
can make an important contribution if it is used as the raw materials out 
of which a better educational product is built. 

Our few years of experience with educational development indicates 
that we finally have a process that can bring about real improvements in 
educational practice. Since this approach is new to education, many of 
the products developed to date, such as some of the new mathematics and 



after the course. If these videotapes do not show significant improve- 



ERiC 




10 



science curriculums, have been built using an inadequate R & D process 
in which essential steps^ have often been overlooked. For example, most 
of these curriculum developments initially failed to include sufficient 
training materials to prepare teachers to use the curriculum effectively. 
This failure, of course, is a violation of the R & D principle that a 
product* should include everything needed to use' it effectively in an 
operational situation. The result of this oversight was that such 
curriculums were often badly misused by teachers. However, most of the 
early developers have now gone back and corrected this serious deficiency. 
On the other hand, most of the current development programs such as the 
SCIS Elementary Science Curriculum, the Minicourse, the Southwest Regional 
Laboratory's Reading Program and the IPI Curriculum, are using the R & D 
process more skillfully to build products that are of proven effectiveness 
and that contain everything needed for implementation. 
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Far West Laboratory for Educational Research and Development 

THE 27 STEPS IN THE DEVELOPMENT PROGRAM 
By Walter Borg and Paul Hood 



Procedures 



The procedures established to implement the program stategy for the 
Teacher Education Program are an outgrowth of the Laboratory-wide research 
and development strategy. At the present time, the Teacher Education Pro- 
gram has established 27 specific steps. The number of steps actually 
executed is, of course, dependent on the nature of the product being 
developed. They are as follows: 

A. Research and Information Collecting 

1. Review literature and prepare report. 

B. Planni ng 

2. State the specific objectives or behavioral changes to be 
achieved and plan a tentative course sequence. 

C. Develop Preliminary Form of Product 

3. Prepare scripts for the instructional lessons. 

4. Prepare teacher handbook and evaluation forms for use in the 
microteach evaluation. 

5. Prepare instructional tapes; record, edit and dub. 

6. Prepare model tapes; record, edit, and dub. 

D. Preliminary Field Testing 

7. Conduct preliminary field test in 1 to 3 schools, using 4 
to 12 teachers. 

8. Evaluate results of field test. 

E. Main Product Revision 

9. Revise scripts based on preliminary field-test results. 

10. Revise handbook; and evaluation forms and print for main 
field test. 

11. Revise instructional tapes; record, edit, and dub. 

12. Revise model tapes; record, edit, and dub. 

13. Prepare follow-up package to be used by teachers during 
nin months completion of the course. 
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F. Main Field Testing 

14. Conduct field test using a sample of 30-75 teachers. 

15. Collect pre-course tapes and post-course tapes of the 
classroom behavior of teachers participating. 

16. Collect delayed post-course tapes of participating 
teachers from four to six months after completing the 
course. 

17. Evaluate main field-test results to determine if the 
course meets the specific behavioral criteria established 
for the course. 

18. Distribute the evaluate follow-up package. 

G. Operational Product Revision 

19. Revise course for operational field test. 

20. Prepare complete implementation package including all 
material needed by $i school to conduct the course without 
outside help. 

H. Operational Field Testing 

21. Train operational test coordinators. 

22. Conduct operational field test. 

23. Evaluate operational field-test results. 

I . Final Product Revisions 

24. Make final revisions in the minicourse prior to mass 
distribution of the course for operational inservice 
use in the schools. 

J. Dissemination and Distribution 



25. Disseminate and distribute course for use. 



K. Report Preparation 

26. Prepare and distribute research and development report, 
giving results of all field testing of the mini course. 

L. Implementati on 

27. Implement course in the schools. 



IZ. 
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